Object Gateway Guide

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Enable the` rgw`module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable rgw

3. Bootstrap the Ceph Object Gateway realm using either the command-line or the yaml specification file:

   • Option 1: Use the command-line interface:

     Syntax

     ceph rgw realm bootstrap [--realm name REALM_NAME] [--zonegroup-name ZONEGROUP_NAME] [--zone-name ZONE_NAME] [--port PORT_NUMBER] [--placement HOSTNAME] [--start-radosgw]

     Example

     [ceph: root@host01 /]# ceph rgw realm bootstrap --realm-name myrealm --zonegroup-name myzonegroup --zone-name myzone --port 5500 --placement="host01 host02" --start-radosgw
     Realm(s) created correctly. Please, use 'ceph rgw realm tokens' to get the token.

   • Option 2: Use the yaml specification file:

     1. As a root user, create the yaml file:

        Syntax

        rgw_realm: REALM_NAME
        rgw_zonegroup: ZONEGROUP_NAME
        rgw_zone: ZONE_NAME
        placement:
          hosts:
           - HOSTNAME_1
           - HOSTNAME_2

        Example

        [root@host01 ~]# cat rgw.yaml
        
        rgw_realm: myrealm
        rgw_zonegroup: myzonegroup
        rgw_zone: myzone
        placement:
          hosts:
           - host01
           - host02

     2. Mount the YAML file under a directory in the container:

        Example

        [root@host01 ~]# cephadm shell --mount rgw.yaml:/var/lib/ceph/rgw/rgw.yaml

     3. Bootstrap the realm:

        Example

        [ceph: root@host01 /]# ceph rgw realm bootstrap -i /var/lib/ceph/rgw/rgw.yaml

        Note

        The specification file used by the rgw module has the same format as the one used by the orchestrator. Thus, you can provide any orchestration supported Ceph Object Gateway parameters including advanced configuration features such as SSL certificates.

4. List the available tokens:

   Example

   [ceph: root@host01 /]# ceph rgw realm tokens | jq
   
   [
     {
       "realm": "myrealm",
       "token": "ewogICAgInJlYWxtX25hbWUiOiAibXlyZWFsbSIsCiAgICAicmVhbG1faWQiOiAiZDA3YzAwZWYtOTA0MS00ZjZlLTg4MDQtN2Q0MDI0MDU1NmFlIiwKICAgICJlbmRwb2ludCI6ICJodHRwOi8vdm0tMDA6NDMyMSIsCiAgICAiYWNjZXNzX2tleSI6ICI5NTY1VFZSMVFWTExFRzdVNFIxRCIsCiAgICAic2VjcmV0IjogImQ3b0FJQXZrNEdYeXpyd3Q2QVZ6bEZNQmNnRG53RVdMMHFDenE3cjUiCn1="
     }
   ]

   Note

   If you run the above command before the Ceph Object Gateway daemons get deployed, it displays a message that there are no tokens as there are no endpoints yet.

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Enable the` rgw`module:

   Example

   [ceph: root@host01 /]# ceph mgr module enable rgw

3. Bootstrap the Ceph Object Gateway realm using either the command-line or the yaml specification file:

   • Option 1: Use the command-line interface:

     Syntax

     ceph rgw realm bootstrap [--realm name REALM_NAME] [--zonegroup-name ZONEGROUP_NAME] [--zone-name ZONE_NAME] [--port PORT_NUMBER] [--placement HOSTNAME] [--start-radosgw]

     Example

     [ceph: root@host01 /]# ceph rgw realm bootstrap --realm-name myrealm --zonegroup-name myzonegroup --zone-name myzone --port 5500 --placement="host01 host02" --start-radosgw
     Realm(s) created correctly. Please, use 'ceph rgw realm tokens' to get the token.

   • Option 2: Use the yaml specification file:

     1. As a root user, create the yaml file:

        Syntax

        rgw_realm: REALM_NAME
        rgw_zonegroup: ZONEGROUP_NAME
        rgw_zone: ZONE_NAME
        placement:
          hosts:
           - HOSTNAME_1
           - HOSTNAME_2
        spec:
          rgw_frontend_port: PORT_NUMBER
        zone_endpoints: http://RGW_HOSTNAME_1:RGW_PORT_NUMBER_1, http://RGW_HOSTNAME_2:RGW_PORT_NUMBER_2

        Example

        [root@host01 ~]# cat rgw.yaml
        
        rgw_realm: myrealm
        rgw_zonegroup: myzonegroup
        rgw_zone: myzone
        placement:
          hosts:
           - host01
           - host02
        spec:
          rgw_frontend_port: 5500
        zone_endpoints: http://<rgw_host1>:<rgw_port1>, http://<rgw_host2>:<rgw_port2>

     2. Mount the YAML file under a directory in the container:

        Example

        [root@host01 ~]# cephadm shell --mount rgw.yaml:/var/lib/ceph/rgw/rgw.yaml

     3. Bootstrap the realm:

        Example

        [ceph: root@host01 /]# ceph rgw realm bootstrap -i /var/lib/ceph/rgw/rgw.yaml

        Note

        The specification file used by the rgw module has the same format as the one used by the orchestrator. Thus, you can provide any orchestration supported Ceph Object Gateway parameters including advanced configuration features such as SSL certificates.

4. List the available tokens:

   Example

   [ceph: root@host01 /]# ceph rgw realm tokens | jq
   
   [
     {
       "realm": "myrealm",
       "token": "ewogICAgInJlYWxtX25hbWUiOiAibXlyZWFsbSIsCiAgICAicmVhbG1faWQiOiAiZDA3YzAwZWYtOTA0MS00ZjZlLTg4MDQtN2Q0MDI0MDU1NmFlIiwKICAgICJlbmRwb2ludCI6ICJodHRwOi8vdm0tMDA6NDMyMSIsCiAgICAiYWNjZXNzX2tleSI6ICI5NTY1VFZSMVFWTExFRzdVNFIxRCIsCiAgICAic2VjcmV0IjogImQ3b0FJQXZrNEdYeXpyd3Q2QVZ6bEZNQmNnRG53RVdMMHFDenE3cjUiCn1="
     }
   ]

   Note

   If you run the above command before the Ceph Object Gateway daemons get deployed, it displays a message that there are no tokens as there are no endpoints yet.

5. Create the secondary zone using these tokens and join the existing realms:

   1. As a root user, create the yaml file:

      Example

      [root@host01 ~]# cat zone-spec.yaml
      rgw_zone: my-secondary-zone
      rgw_realm_token: <token>
      placement:
        hosts:
         - ceph-node-1
         - ceph-node-2
      spec:
        rgw_frontend_port: 5500

   2. Mount the zone-spec.yaml file under a directory in the container:

      Example

      [root@host01 ~]# cephadm shell --mount zone-spec.yaml:/var/lib/ceph/radosgw/zone-spec.yaml

   3. Enable the` rgw`module on the secondary zone:

      Example

      [ceph: root@host01 /]# ceph mgr module enable rgw

   4. Create the secondary zone:

      Example

      [ceph: root@host01 /]# ceph rgw zone create -i /var/lib/ceph/radosgw/zone-spec.yaml

Procedure

1. Create a mount point for the NVMe drive.

   Syntax

   mkfs.ext4 nvme-drive-path

   Example

   [ceph: root@host01 /]# mkfs.ext4 /dev/nvme0n1
   mount /dev/nvme0n1 /mnt/nvme0n1/

2. Create a cache directory path.

   Syntax

   mkdir <nvme-mount-path>/cache-directory-name

   Example

   [ceph: root@host01 /]# mkdir /mnt/nvme0n1/rgw_datacache

3. Provide a+rwx permission to nvme-mount-path and rgw_d3n_l1_datacache_persistent_path.

   Syntax

   chmod a+rwx nvme-mount-path ; chmod a+rwx rgw_d3n_l1_datacache_persistent_path

   Example

   [ceph: root@host01 /]# chmod a+rwx /mnt/nvme0n1 ; chmod a+rwx /mnt/nvme0n1/rgw_datacache/

4. Create/Modify a RGW specification file with extra_container_args to add rgw_d3n_l1_datacache_persistent_path into podman unit.run.

   Syntax

   "extra_container_args:
             "-v"
             "rgw_d3n_l1_datacache_persistent_path:rgw_d3n_l1_datacache_persistent_path"
   			"

   Example

   [ceph: root@host01 /]# cat rgw-spec.yml
   	service_type: rgw
   	service_id: rgw.test
   	placement:
   		hosts:
   		    host1
   		    host2
   		extra_container_args:
   		    "-v"
   		    "/mnt/nvme0n1/rgw_datacache/:/mnt/nvme0n1/rgw_datacache/"

   Note

   If there are multiple instances of RGW in a single host, then a separate rgw_d3n_l1_datacache_persistent_path has to be created for each instance and add each path in extra_container_args.

   Example:

   For two instances of RGW in each host, create two separate cache-directory under rgw_d3n_l1_datacache_persistent_path: /mnt/nvme0n1/rgw_datacache/rgw1 and /mnt/nvme0n1/rgw_datacache/rgw2

   Example for "extra_container_args" in rgw specification file:

   "extra_container_args:
   	"-v"
   	"/mnt/nvme0n1/rgw_datacache/rgw1/:/mnt/nvme0n1/rgw_datacache/rgw1/"
   	"-v"
   	"/mnt/nvme0n1/rgw_datacache/rgw2/:/mnt/nvme0n1/rgw_datacache/rgw2/"
   "

   Example for rgw-spec.yml::

   [ceph: root@host01 /]# cat rgw-spec.yml
   	service_type: rgw
   		service_id: rgw.test
   		placement:
   			hosts:
   				host1
   				host2
   			count_per_host: 2
   			extra_container_args:
   				"-v"
   				"/mnt/nvme0n1/rgw_datacache/rgw1/:/mnt/nvme0n1/rgw_datacache/rgw1/"
   				"-v"
   				"/mnt/nvme0n1/rgw_datacache/rgw2/:/mnt/nvme0n1/rgw_datacache/rgw2/"

5. Redeploy the RGW service:

   Example

   [ceph: root@host01 /]# ceph orch apply -i rgw-spec.yml

Example procedure

1. Create a test object:

   Note

   The test object needs to be larger than 4 MB to cache.

   Example

   [ceph: root@host01 /]# fallocate -l 1G ./1G.dat
   [ceph: root@host01 /]# s3cmd mb s3://bkt
   [ceph: root@host01 /]# s3cmd put ./1G.dat s3://bkt

2. Perform GET of an object:

   Example

   [ceph: root@host01 /]# s3cmd get s3://bkt/1G.dat /dev/shm/1G_get.dat
   download: 's3://bkt/1G.dat' -> './1G_get.dat'  [1 of 1]
   1073741824 of 1073741824   100% in   13s    73.94 MB/s  done

3. Verify cache creation. Cache will be created with the name consisting of object key-name within a configured rgw_d3n_l1_datacache_persistent_path.

   Example

   [ceph: root@host01 /]# ls -lh /mnt/nvme/rgw_datacache
   
   rw-rr. 1 ceph ceph 1.0M Jun  2 06:18 cc7f967c-0021-43b2-9fdf-23858e868663.615391.1_shadow.ZCiCtMWeu_19wb100JIEZ-o4tv2IyA_1

4. Once the cache is created for an object, the next GET operation for that object will access from cache resulting in faster access.

   Example

   [ceph: root@host01 /]# s3cmd get s3://bkt/1G.dat /dev/shm/1G_get.dat
   download: 's3://bkt/1G.dat' -> './1G_get.dat'  [1 of 1]
   1073741824 of 1073741824   100% in    6s   155.07 MB/s  done

   In the above example, to demonstrate the cache acceleration, we are writing to RAM drive (/dev/shm).

1. S3 static web hosting uses Ceph Object Gateway instances that are separate and distinct from instances used for standard S3/Swift API use cases.
2. Gateway instances hosting S3 static web sites should have separate, non-overlapping domain names from the standard S3/Swift API gateway instances.
3. Gateway instances hosting S3 static web sites should use separate public-facing IP addresses from the standard S3/Swift API gateway instances.
4. Gateway instances hosting S3 static web sites load balance, and if necessary terminate SSL, using HAProxy/keepalived.

1. Create an S3 bucket. The bucket name MIGHT be the same as the website’s domain name. For example, mysite.com may have a bucket name of mysite.com. This is required for AWS, but it is NOT required for Ceph.

   • See the Static web hosting DNS configuration section in the Red Hat Ceph Storage Object Gateway Guide for details.
2. Upload the static website content to the bucket. Contents may include HTML, CSS, client-side JavaScript, images, audio/video content, and other downloadable files. A website MUST have an index.html file and might have an error.html file.
3. Verify the website’s contents. At this point, only the creator of the bucket has access to the contents.
4. Set permissions on the files so that they are publicly readable.

Procedure

1. Create a new ingress.yaml file:

   Example

   [root@host01 ~] touch ingress.yaml

2. Open the ingress.yaml file for editing. Added the following options, and add values applicable to the environment:

   Syntax

   service_type: ingress 1
   service_id: SERVICE_ID 2
   placement: 3
     hosts:
       - HOST1
       - HOST2
       - HOST3
   spec:
     backend_service: SERVICE_ID
     virtual_ip: IP_ADDRESS/CIDR 4
     frontend_port: INTEGER 5
     monitor_port: INTEGER 6
     virtual_interface_networks: 7
       - IP_ADDRESS/CIDR
     ssl_cert: | 8

   1

   Must be set to ingress.

   2

   Must match the existing Ceph Object Gateway service name.

   3

   Where to deploy the haproxy and keepalived containers.

   4

   The virtual IP address where the ingress service is available.

   5

   The port to access the ingress service.

   6

   The port to access the haproxy load balancer status.

   7

   Optional list of available subnets.

   8

   Optional SSL certificate and private key.

   Example

   service_type: ingress
   service_id: rgw.foo
   placement:
     hosts:
       - host01.example.com
       - host02.example.com
       - host03.example.com
   spec:
     backend_service: rgw.foo
     virtual_ip: 192.168.1.2/24
     frontend_port: 8080
     monitor_port: 1967
     virtual_interface_networks:
       - 10.10.0.0/16
     ssl_cert: |
       -----BEGIN CERTIFICATE-----
       MIIEpAIBAAKCAQEA+Cf4l9OagD6x67HhdCy4Asqw89Zz9ZuGbH50/7ltIMQpJJU0
       gu9ObNtIoC0zabJ7n1jujueYgIpOqGnhRSvsGJiEkgN81NLQ9rqAVaGpadjrNLcM
       bpgqJCZj0vzzmtFBCtenpb5l/EccMFcAydGtGeLP33SaWiZ4Rne56GBInk6SATI/
       JSKweGD1y5GiAWipBR4C74HiAW9q6hCOuSdp/2WQxWT3T1j2sjlqxkHdtInUtwOm
       j5Ism276IndeQ9hR3reFR8PJnKIPx73oTBQ7p9CMR1J4ucq9Ny0J12wQYT00fmJp
       -----END CERTIFICATE-----
       -----BEGIN PRIVATE KEY-----
       MIIEBTCCAu2gAwIBAgIUGfYFsj8HyA9Zv2l600hxzT8+gG4wDQYJKoZIhvcNAQEL
       BQAwgYkxCzAJBgNVBAYTAklOMQwwCgYDVQQIDANLQVIxDDAKBgNVBAcMA0JMUjEM
       MAoGA1UECgwDUkhUMQswCQYDVQQLDAJCVTEkMCIGA1UEAwwbY2VwaC1zc2wtcmhj
       czUtOGRjeHY2LW5vZGU1MR0wGwYJKoZIhvcNAQkBFg5hYmNAcmVkaGF0LmNvbTAe
       -----END PRIVATE KEY-----

3. Launch the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell --mount ingress.yaml:/var/lib/ceph/radosgw/ingress.yaml

4. Configure the latest haproxy and keepalived images:

   Syntax

   ceph config set mgr mgr/cephadm/container_image_haproxy HAPROXY_IMAGE_ID
   ceph config set mgr mgr/cephadm/container_image_keepalived KEEPALIVED_IMAGE_ID

   Red Hat Enterprise Linux 8

   [ceph: root@host01 /]# ceph config set mgr mgr/cephadm/container_image_haproxy registry.redhat.io/rhceph/rhceph-haproxy-rhel8:latest
   [ceph: root@host01 /]# ceph config set mgr mgr/cephadm/container_image_keepalived registry.redhat.io/rhceph/keepalived-rhel8:latest

   Red Hat Enterprise Linux 9

   [ceph: root@host01 /]# ceph config set mgr mgr/cephadm/container_image_haproxy registry.redhat.io/rhceph/rhceph-haproxy-rhel9:latest
   [ceph: root@host01 /]# ceph config set mgr mgr/cephadm/container_image_keepalived registry.redhat.io/rhceph/keepalived-rhel9:latest

5. Install and configure the new ingress service using the Ceph orchestrator:

   [ceph: root@host01 /]# ceph orch apply -i /var/lib/ceph/radosgw/ingress.yaml

6. After the Ceph orchestrator completes, verify the HA configuration.

   1. On the host running the ingress service, check that the virtual IP address appears:

      Example

      [root@host01 ~]# ip addr show

   2. Try reaching the Ceph Object Gateway from a Ceph client:

      Syntax

      wget HOST_NAME

      Example

      [root@client ~]# wget host01.example.com

      If this returns an index.html with similar content as in the example below, then the HA configuration for the Ceph Object Gateway is working properly.

      Example

      <?xml version="1.0" encoding="UTF-8"?>
      	<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
      		<Owner>
      			<ID>anonymous</ID>
      			<DisplayName></DisplayName>
      		</Owner>
      		<Buckets>
      		</Buckets>
      	</ListAllMyBucketsResult>

Procedure

1. Set the <ArchiveZone> lifecycle policy rule. For more information about creating a lifecycle policy, see See the Creating a lifecycle management policy section in the Red Hat Ceph Storage Object Gateway Guide for more details.

   Example

   <?xml version="1.0" ?>
   <LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
           <Rule>
                   <ID>delete-1-days-az</ID>
                   <Filter>
   		  <Prefix></Prefix>
   		  <ArchiveZone />   1
                   </Filter>
                   <Status>Enabled</Status>
                   <Expiration>
                           <Days>1</Days>
                   </Expiration>
           </Rule>
   </LifecycleConfiguration>

2. Optional: See if a specific lifecycle policy contains an archive zone rule.

   Syntax

   radosgw-admin lc get --bucket BUCKET_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin lc get --bucket test-bkt
   
   {
       "prefix_map": {
           "": {
               "status": true,
               "dm_expiration": true,
               "expiration": 0,
               "noncur_expiration": 2,
               "mp_expiration": 0,
               "transitions": {},
               "noncur_transitions": {}
           }
       },
       "rule_map": [
           {
               "id": "Rule 1",
               "rule": {
                   "id": "Rule 1",
                   "prefix": "",
                   "status": "Enabled",
                   "expiration": {
                       "days": "",
                       "date": ""
                   },
                   "noncur_expiration": {
                       "days": "2",
                       "date": ""
                   },
                   "mp_expiration": {
                       "days": "",
                       "date": ""
                   },
                   "filter": {
                       "prefix": "",
                       "obj_tags": {
                           "tagset": {}
                       },
                       "archivezone": ""   1
                   },
                   "transitions": {},
                   "noncur_transitions": {},
                   "dm_expiration": true
               }
           }
       ]
   }

   1 1

   The archive zone rule. This is an example of a lifecycle policy with an archive zone rule.

3. If the Ceph Object Gateway user is deleted, the buckets at the archive site owned by that user is inaccessible. Link those buckets to another Ceph Object Gateway user to access the data.

   Syntax

    radosgw-admin bucket link --uid NEW_USER_ID --bucket BUCKET_NAME --yes-i-really-mean-it

   Example

   [ceph: root@host01 /]# radosgw-admin bucket link --uid arcuser1 --bucket arc1-deleted-da473fbbaded232dc5d1e434675c1068 --yes-i-really-mean-it

Procedure

1. Create or update a directional sync flow. To create or update directional sync flow for a specific bucket, use the --bucket option.

   Syntax

   radosgw-admin sync group flow create --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=directional --source-zone=SOURCE_ZONE --dest-zone=DESTINATION_ZONE

2. Create or update a symmetrical sync flow. To specify multiple zones for a symmetrical flow type, use a comma-separated list for the --zones option.

   Syntax

   radosgw-admin sync group flow create --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=symmetrical --zones=ZONE_NAME

Procedure

1. Remove a directional sync flow. To remove the directional sync flow for a specific bucket, use the --bucket option.

   Syntax

   radosgw-admin sync group flow remove --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=directional --source-zone=SOURCE_ZONE --dest-zone=DESTINATION_ZONE

2. Remove specific zones from a symmetrical sync flow. To remove multiple zones from a symmetrical flow, use a comma-separated list for the --zones option.

   Syntax

   radosgw-admin sync group flow remove --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=symmetrical --zones=ZONE_NAME

3. Remove a symmetrical sync flow. To remove the sync flow from a bucket, use the --bucket option.

   Syntax

   radosgw-admin sync group flow remove --bucket=BUCKET_NAME --group-id=GROUP_ID --flow-id=FLOW_ID --flow-type=symmetrical --zones=ZONE_NAME

Procedure

1. Create a new sync group with the status set to allowed.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --group-id=group1 --status=allowed

   Note

   Until a fully configured zonegroup replication policy is created, it is recommended to set the --status to allowed, to prevent the replication from starting.

2. Create a flow policy for the newly created group with the --flow-type set as symmetrical to enable bi-directional replication.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group flow create --group-id=group1 \
                           --flow-id=flow-mirror --flow-type=symmetrical \
                           --zones=us-east,us-west

3. Create a new pipe called pipe.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --group-id=group1 \
                             --pipe-id=pipe1 --source-zones='*' \
                             --source-bucket='*' --dest-zones='*' \
                             --dest-bucket='*'

   Note

   Use the * wildcard for zones to include all zones set in the previous flow policy, and * for buckets to replicate all existing buckets in the zones.

4. After configuring the bucket sync policy, set the --status to enabled.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group modify --group-id=group1 --status=enabled

5. Update and commit the new period.

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

   Note

   Updating and committing the period is mandatory for a zonegroup policy.

6. Optional: Execute sync info --bucket=bucket_name command to check the sync source, and destination for a specific bucket. All buckets in zones us-east and us-west replicates bi-directionally.

   Example

   [ceph: root@host01 /]# radosgw-admin sync info --bucket=buck
   {
       "sources": [
           {
               "id": "pipe1",
               "source": {
                   "zone": "us-west",
                   "bucket": "buck:115b12b3-....4409.1"
               },
               "dest": {
                   "zone": "us-east",
                   "bucket": "buck:115b12b3-....4409.1"
               },
               "params": {
   ...
               }
           }
       ],
       "dests": [
           {
               "id": "pipe1",
               "source": {
                   "zone": "us-east",
                   "bucket": "buck:115b12b3-....4409.1"
               },
               "dest": {
                   "zone": "us-west",
                   "bucket": "buck:115b12b3-....4409.1"
               },
              ...
           }
       ],
       ...
   }

   The id field in the above output reflects the pipe rule that generated that entry. A single rule can generate multiple sync entries as seen in the below example.

7. Optional: Run the sync info command to retrieve information about the expected bucket sync sources and targets, as defined in the policy.

Procedure

1. Set the zonegroup policy --status to allowed to permit per-bucket replication.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group modify --group-id=group1 --status=allowed

2. Update the period after modifying the zonegroup policy.

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

3. Create a sync group for the bucket we want to synchronize to and set --status to enabled.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group create --bucket=buck \
                             --group-id=buck-default --status=enabled

4. Create a pipe for the group that was created in the previous step.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group pipe create --bucket=buck \
                             --group-id=buck-default --pipe-id=pipe1 \
                             --source-zones='*' --dest-zones='*'

   Note

   Use wildcards * to specify the source and destination zones for the bucket replication.

5. Optional: To retrieve information about the expected bucket sync sources and targets, as defined by the sync policy, run the radosgw-admin bucket sync info command with the --bucket flag.

   Example

   [ceph: root@host01 /]# radosgw-admin bucket sync info --bucket buck
   realm 33157555-f387-44fc-b4b4-3f9c0b32cd66 (india)
   zonegroup 594f1f63-de6f-4e1e-90b6-105114d7ad55 (shared)
   zone ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5 (primary)
   bucket :buck[ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1]
   
   source zone e0e75beb-4e28-45ff-8d48-9710de06dcd0
   bucket :buck[ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1]

6. Optional: To retrieve information about the expected sync sources and targets, as defined by the sync policy, run the radosgw-admin sync info command with the --bucket flag.

   Example

   [ceph: root@host01 /]# radosgw-admin sync info --bucket buck
   {
       "id": "pipe1",
       "source": {
           "zone": "secondary",
           "bucket": "buck:ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1"
       },
       "dest": {
           "zone": "primary",
           "bucket": "buck:ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1"
       },
       "params": {
           "source": {
               "filter": {
                   "tags": []
               }
           },
           "dest": {},
           "priority": 0,
           "mode": "system",
           "user": ""
       }
   },
       {
       "id": "pipe1",
       "source": {
           "zone": "primary",
           "bucket": "buck:ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1"
       },
       "dest": {
           "zone": "secondary",
           "bucket": "buck:ffaa5ba4-c1bd-4c17-b176-2fe34004b4c5.16191.1"
       },
       "params": {
           "source": {
               "filter": {
                   "tags": []
               }
           },
           "dest": {},
           "priority": 0,
           "mode": "system",
           "user": ""
       }
   }

Procedure

1. Run the sync group modify command to change the status from allowed to forbidden to interrupt replication of the bucket between zones us-east and us-west.

   Example

   [ceph: root@host01 /]# radosgw-admin sync group modify --group-id  buck-default --status forbidden --bucket buck
   {
       "groups": [
           {
               "id": "buck-default",
               "data_flow": {},
               "pipes": [
                   {
                       "id": "pipe1",
                       "source": {
                           "bucket": "*",
                           "zones": [
                               "*"
                           ]
                       },
                       "dest": {
                           "bucket": "*",
                           "zones": [
                               "*"
                           ]
                       },
                       "params": {
                           "source": {
                               "filter": {
                                   "tags": []
                               }
                           },
                           "dest": {},
                           "priority": 0,
                           "mode": "system",
                       }
                   }
               ],
               "status": "forbidden"
           }
       ]
   }

   Note

   No update and commit for the period is required as this is a bucket sync policy.

2. Optional: Run sync info command command to check the status of the sync for bucket buck.

   Example

   [ceph: root@host01 /]# radosgw-admin sync info --bucket buck
   {
       "sources": [],
       "dests": [],
       "hints": {
           "sources": [],
           "dests": []
       },
       "resolved-hints-1": {
           "sources": [],
           "dests": []
       },
       "resolved-hints": {
           "sources": [],
           "dests": []
       }
   }
   
   Sync is disabled for bucket buck

   Note

   There are no source and destination targets as the replication has been interrupted.

1. Extract/Download a PEM-formatted certificate for the Certificate Authority (CA) that signed the LDAP server’s SSL certificate.
2. Confirm that /etc/openldap/ldap.conf does not have TLS_REQCERT set.
3. Confirm that /etc/openldap/ldap.conf contains a TLS_CACERTDIR /etc/openldap/certs setting.

4. Use the certutil command to add the AD CA to the store at /etc/openldap/certs. For example, if the CA is "msad-frog-MSAD-FROG-CA", and the PEM-formatted CA file is ldap.pem, use the following command:

   Example

   # certutil -d /etc/openldap/certs -A -t "TC,," -n "msad-frog-MSAD-FROG-CA" -i /path/to/ldap.pem

5. Update SELinux on all remote LDAP sites:

   Example

   # setsebool -P httpd_can_network_connect on

   Note

   This still has to be set even if SELinux is in permissive mode.

6. Make the certs database world-readable:

   Example

   # chmod 644 /etc/openldap/certs/*

7. Connect to the server using the "ldapwhoami" command as a non-root user.

   Example

   $ ldapwhoami -H ldaps://rh-directory-server.example.com -d 9

   The -d 9 option will provide debugging information in case something went wrong with the SSL negotiation.

Procedure

1. Create an LDAP user for the Ceph Object Gateway, and make a note of the binddn. Since the Ceph object gateway uses the ceph user, consider using ceph as the username. The user needs to have permissions to search the directory. The Ceph Object Gateway binds to this user as specified in rgw_ldap_binddn.

2. Test to ensure that the user creation worked. Where ceph is the user ID under People and example.com is the domain, you can perform a search for the user.

   # ldapsearch -x -D "uid=ceph,ou=People,dc=example,dc=com" -W -H ldaps://example.com -b "ou=People,dc=example,dc=com" -s sub 'uid=ceph'

3. On each gateway node, create a file for the user’s secret. For example, the secret might get stored in a file entitled /etc/bindpass. For security, change the owner of this file to the ceph user and group to ensure it is not globally readable.

4. Add the rgw_ldap_secret option:

   Syntax

   ceph config set client.rgw OPTION VALUE

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_ldap_secret /etc/bindpass

5. Patch the bind password file to the Ceph Object Gateway container and reapply the Ceph Object Gateway specification:

   Example

   service_type: rgw
   service_id: rgw.1
   service_name: rgw.rgw.1
   placement:
     label: rgw
     extra_container_args:
     - -v
     - /etc/bindpass:/etc/bindpass

   Note

   /etc/bindpass is not shipped automatically with Red Hat Ceph Storage and you need to ensure that the content is available on all the possible Ceph Object Gateway instance nodes.

1. Specifying a partial filter:

   Example

   "objectclass=inetorgperson"

   The Ceph Object Gateway generates the search filter with the user name from the token and the value of rgw_ldap_dnattr. The constructed filter is then combined with the partial filter from the rgw_ldap_searchfilter value. For example, the user name and the settings generate the final search filter:

   Example

   "(&(uid=joe)(objectclass=inetorgperson))"

   User joe is only granted access if he is found in the LDAP directory, has an object class of inetorgperson, and specifies a valid password.

2. Specifying a complete filter:

   A complete filter must contain a USERNAME token which is substituted with the user name during the authentication attempt. The rgw_ldap_dnattr setting is not used in this case. For example, to limit valid users to a specific group, use the following filter:

   Example

   "(&(uid=@USERNAME@)(memberOf=cn=ceph-users,ou=groups,dc=mycompany,dc=com))"

1. Export the access key:

   Syntax

   export RGW_ACCESS_KEY_ID="USERNAME"

2. Export the secret key:

   Syntax

   export RGW_SECRET_ACCESS_KEY="PASSWORD"

3. Export the token. For LDAP, use ldap as the token type (ttype).

   Example

   radosgw-token --encode --ttype=ldap

   For Active Directory, use ad as the token type.

   Example

   radosgw-token --encode --ttype=ad

   The result is a base-64 encoded string, which is the access token. Provide this access token to S3 clients in lieu of the access key. The secret key is no longer required.

4. Optional: For added convenience, export the base-64 encoded string to the RGW_ACCESS_KEY_ID environment variable if the S3 client uses the environment variable.

   Example

   export RGW_ACCESS_KEY_ID="ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogImNlcGgiLAogICAgICAgICJrZXkiOiAiODAwI0dvcmlsbGEiCiAgICB9Cn0K"

Procedure.

1. Use the RGW_ACCESS_KEY_ID environment variable to configure the Ceph Object Gateway client. Alternatively, you can copy the base-64 encoded string and specify it as the access key. Following is an example of the configured S3 client:

   Example

   cat .aws/credentials
   
   [default]
   aws_access_key_id = ewogICaGbnjlwe9UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogInBhc3M0Q2VwaCIKICAgIH0KfQo=
   aws_secret_access_key =

   Note

   The secret key is no longer required.

2. Run the aws s3 ls command to verify the user:

   Example

   [root@host01 ~]# aws s3 ls --endpoint http://host03
   
   2023-12-11 17:08:50 mybucket
   2023-12-24 14:55:44 mybucket2

3. Optional: You can also run the radosgw-admin user command to verify the user in the directory:

   Example

   [root@host01 ~]# radosgw-admin user info --uid dir1
   {
       "user_id": "dir1",
       "display_name": "dir1",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "subusers": [],
       "keys": [],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "default_storage_class": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "ldap",
       "mfa_ids": []
   }

Procedure

1. Create an LDAP user for the Ceph Object Gateway, and make a note of the binddn. Since the Ceph object gateway uses the ceph user, consider using ceph as the username. The user needs to have permissions to search the directory. The Ceph Object Gateway binds to this user as specified in rgw_ldap_binddn.

2. Test to ensure that the user creation worked. Where ceph is the user ID under People and example.com is the domain, you can perform a search for the user.

   # ldapsearch -x -D "uid=ceph,ou=People,dc=example,dc=com" -W -H ldaps://example.com -b "ou=People,dc=example,dc=com" -s sub 'uid=ceph'

3. On each gateway node, create a file for the user’s secret. For example, the secret might get stored in a file entitled /etc/bindpass. For security, change the owner of this file to the ceph user and group to ensure it is not globally readable.

4. Add the rgw_ldap_secret option:

   Syntax

   ceph config set client.rgw OPTION VALUE

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_ldap_secret /etc/bindpass

5. Patch the bind password file to the Ceph Object Gateway container and reapply the Ceph Object Gateway specification:

   Example

   service_type: rgw
   service_id: rgw.1
   service_name: rgw.rgw.1
   placement:
     label: rgw
     extra_container_args:
     - -v
     - /etc/bindpass:/etc/bindpass

   Note

   /etc/bindpass is not shipped automatically with Red Hat Ceph Storage and you need to ensure that the content is available on all the possible Ceph Object Gateway instance nodes.

1. Export the access key:

   Syntax

   export RGW_ACCESS_KEY_ID="USERNAME"

2. Export the secret key:

   Syntax

   export RGW_SECRET_ACCESS_KEY="PASSWORD"

3. Export the token. For LDAP, use ldap as the token type (ttype).

   Example

   radosgw-token --encode --ttype=ldap

   For Active Directory, use ad as the token type.

   Example

   radosgw-token --encode --ttype=ad

   The result is a base-64 encoded string, which is the access token. Provide this access token to S3 clients in lieu of the access key. The secret key is no longer required.

4. Optional: For added convenience, export the base-64 encoded string to the RGW_ACCESS_KEY_ID environment variable if the S3 client uses the environment variable.

   Example

   export RGW_ACCESS_KEY_ID="ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogImNlcGgiLAogICAgICAgICJrZXkiOiAiODAwI0dvcmlsbGEiCiAgICB9Cn0K"

Procedure.

1. Use the RGW_ACCESS_KEY_ID environment variable to configure the Ceph Object Gateway client. Alternatively, you can copy the base-64 encoded string and specify it as the access key. Following is an example of the configured S3 client:

   Example

   cat .aws/credentials
   
   [default]
   aws_access_key_id = ewogICaGbnjlwe9UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogInBhc3M0Q2VwaCIKICAgIH0KfQo=
   aws_secret_access_key =

   Note

   The secret key is no longer required.

2. Run the aws s3 ls command to verify the user:

   Example

   [root@host01 ~]# aws s3 ls --endpoint http://host03
   
   2023-12-11 17:08:50 mybucket
   2023-12-24 14:55:44 mybucket2

3. Optional: You can also run the radosgw-admin user command to verify the user in the directory:

   Example

   [root@host01 ~]# radosgw-admin user info --uid dir1
   {
       "user_id": "dir1",
       "display_name": "dir1",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "subusers": [],
       "keys": [],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "default_storage_class": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "ldap",
       "mfa_ids": []
   }

Procedure

1. List the endpoints under the Swift service:

   [root@swift~]# openstack endpoint list --service=swift

2. Verify settings for the endpoints listed in the previous command:

   Syntax

   [root@swift~]# openstack endpoint show ENDPOINT_ID

   Showing the endpoints will echo the endpoints settings, and the service settings.

   Table 6.2. Example

   Field	Value
   adminurl	http://radosgw.example.com:8080/swift/v1
   enabled	True
   id	e4249d2b60e44743a67b5e5b38c18dd3
   internalurl	http://radosgw.example.com:8080/swift/v1
   publicurl	http://radosgw.example.com:8080/swift/v1
   region	us-west
   service_id	37c4c0e79571404cb4644201a4a6e5ee
   service_name	swift
   service_type	object-store

Procedure

1. Convert the OpenSSL certificate to the nss db format:

   Example

   [root@osp ~]# mkdir /var/ceph/nss
   
   [root@osp ~]# openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | \
       certutil -d /var/ceph/nss -A -n ca -t "TCu,Cu,Tuw"
   
   [root@osp ~]# openssl x509 -in /etc/keystone/ssl/certs/signing_cert.pem -pubkey | \
       certutil -A -d /var/ceph/nss -n signing_cert -t "P,P,P"

2. Install Keystone’s SSL certificate in the node running the Ceph Object Gateway. Alternatively set the value of the configurable rgw_keystone_verify_ssl setting to false.

   Setting rgw_keystone_verify_ssl to false means that the gateway will not attempt to verify the certificate.

Procedure

1. Do the following for each gateway instance.

   1. Set the nss_db_path setting to the path where the NSS database is stored:

      Example

      [ceph: root@host01 /]# ceph config set client.rgw nss_db_path "/var/lib/ceph/radosgw/ceph-rgw.rgw01/nss"

2. Provide authentication credentials:

   It is possible to configure a Keystone service tenant, user, and password for keystone for the OpenStack Identity API, similar to the way system administrators tend to configure OpenStack services. Providing a username and password avoids providing the shared secret to the rgw_keystone_admin_token setting.

   Important

   Red Hat recommends disabling authentication by admin token in production environments. The service tenant credentials should have admin privileges.

   The necessary configuration options are:

   Syntax

   ceph config set client.rgw rgw_keystone_verify_ssl TRUE/FALSE
   ceph config set client.rgw rgw_s3_auth_use_keystone TRUE/FALSE
   ceph config set client.rgw rgw_keystone_api_version API_VERSION
   ceph config set client.rgw rgw_keystone_url KEYSTONE_URL:ADMIN_PORT
   ceph config set client.rgw rgw_keystone_accepted_roles ACCEPTED_ROLES_
   ceph config set client.rgw rgw_keystone_accepted_admin_roles ACCEPTED_ADMIN_ROLES
   ceph config set client.rgw rgw_keystone_admin_domain default
   ceph config set client.rgw rgw_keystone_admin_project SERVICE_NAME
   ceph config set client.rgw rgw_keystone_admin_user KEYSTONE_TENANT_USER_NAME
   ceph config set client.rgw rgw_keystone_admin_password KEYSTONE_TENANT_USER_PASSWORD
   ceph config set client.rgw rgw_keystone_implicit_tenants KEYSTONE_IMPLICIT_TENANT_NAME
   ceph config set client.rgw rgw_swift_versioning_enabled TRUE/FALSE
   ceph config set client.rgw rgw_swift_enforce_content_length TRUE/FALSE
   ceph config set client.rgw rgw_swift_account_in_url TRUE/FALSE
   ceph config set client.rgw rgw_trust_forwarded_https TRUE/FALSE
   ceph config set client.rgw rgw_max_attr_name_len MAXIMUM_LENGTH_OF_METADATA_NAMES
   ceph config set client.rgw rgw_max_attrs_num_in_req MAXIMUM_NUMBER_OF_METADATA_ITEMS
   ceph config set client.rgw rgw_max_attr_size MAXIMUM_LENGTH_OF_METADATA_VALUE
   ceph config set client.rgw rgw_keystone_accepted_reader_roles SwiftSystemReader

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_verify_ssl false
   [ceph: root@host01 /]# ceph config set client.rgw rgw_s3_auth_use_keystone true
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_api_version 3
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_url http://<public Keystone endpoint>:5000/
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_accepted_roles 'member, Member, admin'
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_accepted_admin_roles 'ResellerAdmin, swiftoperator'
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_admin_domain default
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_admin_project service
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_admin_user swift
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_admin_password password
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_implicit_tenants true
   [ceph: root@host01 /]# ceph config set client.rgw rgw_swift_versioning_enabled  true
   [ceph: root@host01 /]# ceph config set client.rgw rgw_swift_enforce_content_length true
   [ceph: root@host01 /]# ceph config set client.rgw rgw_swift_account_in_url true
   [ceph: root@host01 /]# ceph config set client.rgw rgw_trust_forwarded_https true
   [ceph: root@host01 /]# ceph config set client.rgw rgw_max_attr_name_len 128
   [ceph: root@host01 /]# ceph config set client.rgw rgw_max_attrs_num_in_req 90
   [ceph: root@host01 /]# ceph config set client.rgw rgw_max_attr_size  1024
   [ceph: root@host01 /]# ceph config set client.rgw rgw_keystone_accepted_reader_roles SwiftSystemReader

   A Ceph Object Gateway user is mapped into a Keystone tenant. A Keystone user has different roles assigned to it on possibly more than a single tenant. When the Ceph Object Gateway gets the ticket, it looks at the tenant, and the user roles that are assigned to that ticket, and accepts or rejects the request according to the rgw_keystone_accepted_roles configurable.